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ABSTRACT 

According to education pundits, traditional-age college students are “digital natives” inherently savvy in digital tech¬ 
nology due to their constant exposure to technology from an early age. This widely held meme is at odds with obser¬ 
vation in the college classroom. In this research, college students in an introductory information technology course 
are surveyed prior to instruction regarding perceptions of their own technological expertise in hardware, software, 
networking, research, computer graphics, computer security, Microsoft Word, Microsoft Excel, and Microsoft Power¬ 
Point. Students are then given pre-tests to assess their expertise in these topics based on course standards. Data from 
the survey and the pre-tests are analyzed. Although students self assess slightly greater expertise in areas in which 
they actively use computers (Windows, Word, PowerPoint, and Research), they generally do not claim technological 
expertise. Phis lack of expertise is confirmed in pre-tests scores. Data analysis provides no supportfor the “digital na¬ 
tive” meme. 


INTRODUCTION 

Among the several monikers attached to the current gen¬ 
eration of traditional-age students, “digital natives” might 
be the most misleading. Coined by educational futurists 
in the early 2000s, the digital native meme suggests indi¬ 
viduals born after a certain date are naturally fluent with 
modern information technologies (Prensky, 2001). Just as 
small children learn language through constant exposure 
in their preschool years, digital natives supposedly became 
fluent in computing and telecommunications technolo¬ 
gies because these digital technologies were ubiquitous in 
their childhood environment. Thus, these digital natives 
do not need to be taught about technology—they “just 
do it.” In contrast, older generations are not so fortunate. 
They are “digital immigrants” who are always awkward in 
the digital environment, never quite comfortable in the 
foreign land of technology in which they did not grow-up. 

The “digital native” meme has obvious logical flaws (tech¬ 
nology creators were digital immigrants, for example). 
Even so, this meme has proven quite popular in both the 
educational environment and in popular culture. In some 
colleges, it has even guided curriculum decisions. How¬ 
ever, research and deeper investigation does not support 
its veracity (Bennet, Maton, & Kerin, 2008; Brown & 
Czerniewicz, 2010; Brumberger, 2011; Helsper & Eynon, 
2010; Margaryan, Littlejohn, & Vojt, 2011; Ng, 2012; 
O’Neil, 2014). 


Despite being repeatedly discredited, the digital native 
myth persists. But what do the “digital natives” think? 
Having grown-up surrounded by technology and having 
been told their entire lifetimes (by some) they are knowl¬ 
edgeable about technology, do they consider themselves 
technologically adept? How well do they perform when 
confronted with questions or tasks involving technology? 
Students’ self-assessment of their own expertise influences 
their attitude and perceptions about the need to study in 
courses involving technology. 

BACKGROUND 

The National Research Council’s Fluency in Technol¬ 
ogy model (National Research Council, 1999) provides 
the framework for the introductory technology course in 
which this research is conducted. The NRC states fluency 
in information technology is composed of three compo¬ 
nents - contemporary skills (accomplish common tasks 
using today’s technology), foundation concepts (under¬ 
stand basic principles and ideas underlying technology), 
and intellectual abilities (use technology to address com¬ 
plex problems). Course assignments and testing are de¬ 
signed to develop technological fluency among students. 

Course Technology’s SAM Office 2010 training and test¬ 
ing system is used to teach contemporary skills in Micro¬ 
soft Word, Excel, and PowerPoint. SAM’s training mod¬ 
ules provide a conceptual framework for each skill then 
demonstrates the skill online. Students are then required 
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to perform the skill for SAM to provide graded credit. 
Instruction is at the level of Course Technology’s New 
Perspectives series introductory textbook (Shaffer, et ah, 
2011) with a handful of advanced topics in Word and Ex¬ 
cel specifically requested by upper division course instruc¬ 
tors. 

Lecture, classroom activities, readings, and digital videos 
are used to teach concepts about hardware, software, net¬ 
working, research, computer graphics, security as well as 
word processing, spreadsheets, and presentations. Con¬ 
tent in these topics is based on Internet & Core Comput¬ 
ing Certification Global Standard 4 (CertiPort, n.d.) at 
the level of a widely used college technology literacy text¬ 
book (Parsons & Oja, 2011) and augmented based on in¬ 
formation literacy guidelines from the American Library 
Association (n.d.). 

SAM 2010 online tests are used to evaluate skills mastery. 
SAM simulates a screen from Word, Excel, or PowerPoint 
and requires students to complete a defined task (such as 
create a new style in Word or edit a pivot table in Excel). 
SAM allows up to three attempts by the student before 
rendering a pass/fail assessment on that individual task. 
Skills tests in this course contain approximately 50 tasks. 

Concept tests are conducted using the Moodle learning 
management system. Questions are primarily multiple- 
choice with some true-false, matching, and short answer. 
For each concept to be tested, Moodle maintains multiple 
equivalent questions in a test bank. Tests are generated 
individually for each student by assembling randomly se¬ 
lected test bank questions. Thus, each student has a differ¬ 
ent but equivalent test. 

The instructor administers skills and concepts pre-tests 
prior to instruction in this course. The SAM skills pre¬ 
tests are the same tests administered later in the semester 
for course credit. The concepts pre-tests are modified ver¬ 
sions of the course final exam (questions about course-spe¬ 
cific activities found in the final exam are removed from 
the pre-tests). 

METHODOLOGY 

This research study was conducted in three freshman-level 
information technology courses in Fall 2013. There were 
41 students in the study. Most students were first semes¬ 
ter freshman of traditional age and were graduates of sub¬ 
urban high schools. A few students were sophomore or 
juniors transfers. Students not in their late teens were in 
their early-to-mid twenties. 

In the first week of the course, the author administered 
a survey asking students to rate their perceived expertise 
in course topics (see Figure 1). Students were instructed 


to mark the line associated with each topic based on as¬ 
sessment of his or her existing knowledge (Aiken, 1996). 
This graphical rating scale was later measured and student 
responses were recoded to a 0-to-100 scale. 

Prior to course instruction, students completed the con¬ 
cepts pre-test and the SAM 2010 skills pre-tests for Micro¬ 
soft Word, Microsoft Excel, and Microsoft PowerPoint. 

RESULTS 

Figures 2 through 11 provide the mean, standard devia¬ 
tion, median, interquartile range, first quartile, and third 
quartile for each self-assessment and pre-test distribution. 
The r 2 coefficient of determination between each self-as¬ 
sessment and its associated pre-test is provided. Data from 
each distribution are binned and displayed in histograms. 
A scatterplot displays the relationship between each stu¬ 
dent’s self-assessment and pre-test score. Note that Micro¬ 
soft Word, Microsoft Excel, and Microsoft PowerPoint 
have both skills pre-tests and concepts pre-tests and thus 
have two sets of pre-test distributions and graphs. 

At the aggregate level, students assess a modest level of ex¬ 
pertise in using the Microsoft Windows operating system. 

Figure 1 

Self-assessment survey instrument. 
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lower ([*=22). At the individual student level, the corre¬ 
lation between self-assessed knowledge and skills perfor¬ 
mance is slight but correlation between self-assessment 
and the concepts pre-test score is practically non-existent. 

At the aggregate level, students did not profess as much 
expertise in Excel as they did in Word and PowerPoint. 
The aggregate self-assessment distribution was very con¬ 
sistent with aggregate performance on the skills pre-test 
(both with [*=39). Aggregate performance on the concepts 
test was very poor. At the individual level, the correlation 
between self-assessed knowledge and skills performance 
is the second highest in this study (r 2 =0.18). Correlation 
between self-assessment and the concepts pre-test score is 
practically non-existent. 

At the aggregate level, student self-assessed expertise in 
PowerPoint is second only to their perceived expertise in 
Microsoft Word. The means of the self-assessment ([*=61) 
and skills pre-test ([*=60) are consistent, though the stan¬ 
dard deviation and interquartile range for the skills test 
are much smaller. While the performance on the concepts 
pre-test is again much poorer than in the skills pre-test, 
scores for the PowerPoint concepts test are noticeably 
higher than those in Word or Excel. At the individual stu¬ 
dent level, correlation between self-assessment and perfor¬ 
mance is poor. 


Aggregate self-assessment data indicates students have 
little confidence in their hardware knowledge. Perfor¬ 
mance in the concepts pre-test was actually slightly better 
than aggregate self-assessment - likely due to the multi¬ 
ple-choice nature of the pre-test. The bulk of the students 
scored at the level of “guessing” on hardware questions 
(approximately 25%). At the individual student level, the 
correlation between self-assessed knowledge and test per¬ 
formance is zero. 

Aggregate self-assessment for software is very similar to 
the self-assessment for hardware above - students profess 
very little expertise. Performance on the software concepts 
pre-test was better than for the hardware pre-test and ac¬ 
tually better than the self-assessment (though this is likely 
an artifact of a multiple-choice testing environment). At 
the individual level, correlation between self-assessment 
and concepts test performance (r 2 =0.19) is higher than 
any other correlation in the study. 

Despite seemingly constant use of networking and tele¬ 
communications technologies, students generally pro¬ 
fessed little expertise in networking. In the aggregate, con¬ 
cept pre-test scores are consistent with self-assessments. 
There is no correlation between perceived expertise and 
pre-test performance. 


Figure 4 

Microsoft Excel self-assessment, skills, and concepts scores. 


Excel _Mean Std Dev Median IQR IstQ_3rd Q_R-Square 
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Figure 6 

Hardware self-assessment and concepts pre-test scores. 
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At the aggregate level, students self-assess research exper¬ 
tise at a level similar to their self-assessment in Windows 
and indicate a relatively high level of perceived expertise. 
Performance in the concepts pre-test, however, is much 
lower and is lower than performance in other topics with 
less perceived expertise (see hardware and networking). 
At the individual student level, there is no correlation 


between self-assessed knowledge and concepts pre-test 
scores. 

Review of aggregate data suggests most students profess 
little knowledge in the area of computer graphics with a 
few students indicating significant knowledge. Aggregate 
performance on the graphics pre-test is consistent with 
self-assessed knowledge. At the individual student level, 
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Figure 7 

Software self-assessment and concepts pre-test scores. 
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Figure 8 

Networking self-assessment and concepts pre-test scores. 
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Figure 9 

Research self-assessment and concepts pre-test scores. 
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Figure 10 

Graphics self-assessment and concepts pre-test scores. 
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Figure 11 

Security self-assessment and concepts pre-test scores. 
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there is no correlation between self-assessed knowledge 
and concepts pre-test performance. 

Review of aggregate data indicates some perception of 
knowledge in this topic. Concept pre-test scores, howev¬ 
er, indicate the contrary. At the individual student level, 
there is no correlation between self-assessment and con¬ 
cept pre-test performance. 

DISCUSSION 

This research was conducted in a college course setting. 
This context is both a strength and a weakness for the 
study. Compared to experiments conducted outside the 
classroom, students are likely more engaged in the in-class 
survey and pre-tests integral to course instruction. On 
the other hand, the pre-tests are designed as part of an in¬ 


structional process and are not be ideal instruments for a 
research experiment. For example, due to classroom time 
constraints, each topic had only four questions in the con¬ 
cepts pre-test. More questions per topic would have yield¬ 
ed more accurate and more statistically powerful results. 

Another potential limitation to the study is the lack of 
well-understood standards of expertise in each content 
area. What did students think “significant knowledge” 
in these topics meant? Had the instructor provided a list 
of knowledge and skills inherent in the concept and skill 
pre-tests, students would likely have made more accu¬ 
rate self-assessments and correlations with pre-test scores 
would have been higher. However, the vagueness of stan¬ 
dards is consistent with the purpose of the research. Pro¬ 
ponents of the “digital native” meme present no standards 
of knowledge, simply attributing vague expertise to an age 


Journal of Learning in Higher Education 


105 



























Keith R. Nelms 



cohort. This research requires students in that age cohort 
to tap into their own vague sense of knowledge in each of 
these topical areas. 

Figure 12 plots the relationship between self-assessment 
and pre-test scores for each topic’s distribution means. 
Note the marked gaps in plotted points between approxi¬ 
mately 40 and 50 on both axes. The only topics students 
self-assessed above the level of 50 are Windows, Word, 
PowerPoint, and Research. The pre-test scores were above 


50 only for Windows, Word-Skill, and PowerPoint-Skill. 
The remaining topics are below the 40 on both scales 
(with a minor exception for the Software pre-test score 
mean). Student self-assessment for Software, Hardware, 
Network, and Graphics are at 30 or lower. 

These data patterns suggest students self-assess greater ex¬ 
pertise in areas where they are active users. They perceive 
expertise in Windows, PowerPoint, Word and Research 
because they use Windows, PowerPoint, Word, and 
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Google on a frequent basis. Figure 12. Scatterplot of self- 
assessment and pre-test means. 

And—for Windows, PowerPoint skills, and Word skills - 
they score significantly better in the pre-test than in other 
topics. As the Research pre-test questions involve a more 
sophisticated understanding of research processes than 
simply “googling,” the Research pre-test scores were much 
lower than students’ expectations. 

It is important to note that pre-test scores for Word, Ex¬ 
cel, and PowerPoint concepts are significantly lower than 
skills scores for those programs. These skills tests are com¬ 
prised of small tasks and allow three opportunities to 
complete the question correctly before it is counted as in¬ 
correct. In such a trial-and-error testing environment, stu¬ 
dents can often manipulate the software correctly with¬ 
out understanding the underlying concepts. For example, 
a student might modify a pivot table in Excel by finding 
keywords from the question in the Excel ribbon without 
understanding the nature and use of pivot tables (an Excel 
concept). While students benefit from the multiple-choice 
nature of the concepts tests, they are not allowed multiple 
attempts to complete the question correctly. Thus, both 
skills and concept pre-tests likely overstate students’ topic 
knowledge. 

Finally, it should be noted that aggregate data tells a dif¬ 
ferent story than individual data. Aggregate data, partic¬ 
ularly expressed in the histograms of Figures 2 through 
11, suggest varying patterns of student expertise among 
the different course topics. For example, Word and Pow¬ 
erPoint skills pre-test performance histograms look very 
similar to the self-assessment histograms for those top¬ 
ics while the histograms for Research look very differ¬ 
ent. There are real world interpretations to aggregate data 
patterns that are intuitively appealing. However, at the 
individual student level, data tell but one story - there is 
almost no correlation between student self-assessments 
and student pre-test performance. Out of thirteen co¬ 
efficients of determination, only four are 0.10 or higher 
(Word skills, Excel skills, PowerPoint skills, and Software 
concepts) and none of the r2 values reach 0.20. Although 
the aggregate data tell stories, these stories only emerge at 
the group level. 

SUMMARY 

To the extent students in this study represents the “digi¬ 
tal native” population, there is little to support the digital 
native meme. These digital natives do not consider them¬ 
selves knowledgeable in the standard topics found in col¬ 
lege-level technology literacy courses nor are they particu¬ 
larly adept at assessing their own technological expertise. 
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